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ABSTRACT 

This study examined the influence of three different 
combinations of document structures and menu designs on users' 
attitude, performance, and learning in five different search tasks. 
The three types of combinations studied were: (1) an explicit menu 
signaling hierarchical structure where cross-referencing was not 
supported (EXH — explicit and hierarchical); (2) an explicit menu 
signaling hierarchical structure in which cross-referencing 
capability was embedded (EXN — explicit and network); and (3) an 
embedded menu signaling both hierarchical structure and 
cross-referencing capability (EMN — embedded and network). Based upon 
the specificity, complexity, id boundary of the search targets, five 
types of searches were studied — when the target was: simple and fully 
known; simple but not partially known; complex and fully known; 
complex but only partially known; and complex and the condition for 
terminating the sear-h was unclear. Results showed that providing 
cross-reference links in small- or medium-sized online documents can 
improve search accuracy but not efficiency. EXN produces the best 
search accuracy, and EMN encourages in-depth search for tasks that 
are complex and not fully known. Although EXN was best received by 
the subjects, it resulted in a greater sense of getting lost for 
those who used the referential links and backtracking links more 
often. Causes of this disorientation, reasons why EXN and EMN 
produced better performance were discussed and search strategies 
employed by the subjects were investigated. The questionnaire used in 
the study is appended. (Contains 54 references.) (JLB) 
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Abstract 



This study examined the influence of three different combinations of document structures and menu designs 
on users' attitude, performance, and learning in five different search tasks. The three types o' combinations studied 
were: (a) an explicit menu signaling hierarchical structure where cross-referencing was not supported (EXH— explicit 
and hierarchical), (b) an explicit menu signaling hierarchical structure in which cross-referencing capability was 
embedded (EXN-explicit and network), and (c) an embedded menu signaling both hierarchical structure and cross- 
referencing capability (EMN-embedded and network). Based upon the specificity, complexity and boundary of the 
search targets, the five types of searches studied were when the target was: (a) simple and fully known, (b) simple 
but only partially known, (c) complex and fully known, (d) complex but only partially known, and (e) complex and 
the condition for terminating the search was unclear. The results of the study showed that providing cross-reference 
links in small- or ,nedium-sized online documents can improve search accuracy, but not efficiency. EXN can 
produce the best search accuracy in most cases and EMN will encourage in-depth search for tasks that are complex 
and not fully known. Although EXN was best received by the subjects, it resulted in a greater sense of getting lost 
for those who used the referential links and backtracking links more often. The causes of this sense of 
disorientation and die reasons why EXN and EMN produced better performance were discussed, and the search 
strategies employed by the subjects were investigated. 

INTRODUCTION 

The hypertext concept has been embraced and studied by many instructional designers, educational 
psychologists, technical communicators, and cognitive, computer, and information scientists. Educational 
researchers believe that hypertext is an ideal knowledge representation format that makes generative or adaptive 
learning possible (Dede, 1987, 1988; Jonassen, 1986, 1988) and that hypertext is suitable for exploratory learning 
for ill-structured, advanced knowledge domains or literary education (Landow, 1989; Spiro & Jehng, 1990; Spiro, 
Feltovich, Jacobson, & Coulson, 1991). Some see hypertext as a platform for multidisciplinary learning in the 
increasingly complex and growing field of science (Davenport & Cronin, 1990; Marchionini & Shneiderman, 1988). 
Some see hypertext and its extension-hypermedia-as a new CBI (computer-based instruction) authoring 
environment (Park, 1991), and some see a new type of CBI application emerging-hypermedia-assisted instruction, 
or HAI (Heller, 1990). 

However, with all the fervent discussions and studies, it is still very difficult to articulate empirically 
determined guidelines for implementing online instructional documents. There are three reasons: First, the text 
types and reading purposes vary in these studies and sometimes are not clearly classified (Dillon, 1990). While 
many studies focus on information searching, detailed information regarding the complexity and specificity of the 
search tasks are seldom provided. Second, the menu designs of the systems studied are often different and vaguely 
described. Some use what Ben Shneiderman calls "embedded menu," i.e., users select a word or phrase of interest 
from within a paragraph to see more detailed information regarding that word or phrase. Some use the traditional 
menu structure where organizational components are explicitly separated from the content, and selection in context is 
only used for referential or associative links, not organizational relationships. Third, the definition of hypertext as 
non-linear text can be vague too. Can hierarchically organized information bases be classified as hypertext? A 
strictly hierarchically organized text without the support for associative or referential links is far from the ideal of 
hypertext-a network of ideas/concepts connected based on their associative relationships. We often cannot tell if 
some studies provide associative or referential links in addition to organizational links to qualify them as hypertext 
systems. As a corollary to the above problems, it is difficult to conclude from previous studies comparing linear 
and non-linear systems whether hypertext systems are suitable for information searching. Studies comparing 
different hypertext browsing systems also cannot explain what it is that contributes to the difference in user 
performance as too many potentially significant variables are different across these systems (Rada, 1991). 

Information Searching 

People often learn through searching for information when they encounter problems. Jonassen and 
Grabinger (1990) rightly claim that information seeking is "a fundamental learning activity, precursive to many 
others" >. 7). It is also a "pervasive human activity" (Nickerson, 1986) and "a special case of problem-solving' 
(Marchionini, 1989). With computers' capability of massive storage, easy manipulation, and fast retrieval of 
information, more and more documents are put online, and electronic databases have demonstrated their potential as a 
learning resource. Interaction with electronic databases bas become "a matter of the utmost concern for education and 
training" (Cotterell, Ennals, & Briggs, 1988). 
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As problems can be classified into two broad categories, i.e., well-defined and ill-defined, in terms of the 
clarity of the starting point and goal, search targets are different in their complexity and specificity. However, in 
most studies, information searching is often defined as looking for answers to specific and explicit questions. A 
study of users' navigation in a database environment found that database users use databases for purposes more than 
just factual retrieval, and there are five discernibly different searching strategies: scanning, browsing, searching, 
exploring, and wandering (Canter, Rivers, & Storrs, 1985). These five searching strategies are defined as: 

Scanning: covering a large area without depth. 
Browsing: following a path until a goal is achieved. 
Searching: striving to find an explicit goal. 
Exploring: finding out the extent of the information given. 
Wandering: purposeless and unstructured globetrotting. 

This account of search strategies provides a more complete picture that encompasses the wide range of information 
searching activities in which users may be involved under real learning and information searching situations. 
Research has shown that information searchers prefer using search facilities such as keyword search or index (Joseph, 
Steinberg, & Jones, 1989). But when search questions are vague, people tend to resort to the browsing or 
exploring strategies (Marchionini & Shneiderman, 1988). Tonta (1991) has argued that "hyptertext systems are not 
designed for fast and efficient fact retrieval. Rather they support unhurried and informal information searching" (p. 
22). Therefore, when the search task is simple and specific, comparison among different browsing systems is 
actually a comparison of search facilities prov : ded in each system rather than a comparison of the structure or 
interface design. The difference a network-structured system can make cannot be found if the ser rch facilities 
provided are different in the systems compared or if search tasks studied are limited to simple ana specific facts. 

Wright (1990) argues that a decomposition of search tasks is necessary in order to understand the 
imputations of functionalities of and interface designs for hypertext systems. Based on how many elements are 
needed id order o answer a question (i.e., complexity), whether each element is known (i.e., specificity), whether 
each element is specifiable to the computer (i.e., specifiability), and whether there is a clear terminating condition 
for the search task (i.e., boundary), Wright (1990) proposes six different types of search tasks: 

1. Search target simple and fully known; 

2. Search target simple but only partially known; 

3. Search target complex and fully known; 

4. Search target computed from online trade-offs and feedback from the computer; 

5. Search target simple but unspecifiable to a computer; and 

6. Search target unrecognizable for the purposes of tenainating the search, (pp. 176-178) 

Another taxonomy of five criteria was used to classify search questions in an attempt to assess incidental 
learning during information retrieval (Jones, 1989). The five criteria are based v>;i how many concepts/components 
are presented in the question (complexity), how easy it is to determine the major entry point for information access 
(specificity), how easy it is to determine the appropriate label as used in the document (focus), how many articles 
need to be selected in order to find the answer (path), and how easy it is to find the right path to the answer 
(accessibihiy). Marchionini (1989) reported that users tend to simply follow the links and use low cognitive load 
strategies w'len browsing a hypertext system. Will a hypertext system with associative links distract information 
searchers from their search goal (Foss, 1989) and encourage serendipitous findings? Or will it expedite the search? 
Will it rf iult in what Foss (1989) calls the "art museum phenomenon," i.e., learners browse through many different 
parts of the document but have difficulties in forming a coherent understanding or abstraction of what has been seen? 
Or will it help make an unclear search goal clearer? How would this influence a user's sense of control, confidence 
in understanding the material and confidence in using such systems? 

Document Structure 

Hierarchical structure is the most common structure used in online documentation, as well as in many 
computer-based instructional programs. "Hierarchical structures are the most natural structures for organizing levels 
of abstraction" (Conklin, 1987, p. 35). As a matter of fact, online writing started out by imitating conventional 
writing on paper, the goal of which is to create "a perfect hierarchy" (Bolter, 1991). According to Ausubel (1968), 
our semantic networks are arranged hierarchically, and research also shows that people construct hierarchical 
representation of the text they read (van Oijk & Kintsch, 1983). Hierarchy is not only a natural organization for 
texts but also a natural strategy for information searching. The organizational structure of a database is found to be 
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the primary determinant of information-searching performance, as it structures the knowledge as well as defines how 
the knowledge can be accessed (Marchionini & Shneiderman, 1988). Guthrie, Britten, and Barker (1991) propose a 
five-component cognitive model for information searching: (a) goal formation, (b) category selection, (c) extraction, 
(d) integration, and (e) recycling. In a series of studies, category selection was found to be the most crucial and time- 
consuming one among the five (Guthrie, 1988; Guthrie & Dreher, 1990). This model is also found to be adequate 
for both sharp and fuzzy problems, i.e., problems that are clear and specific and problems that are vague (Guthrie & 
Dreher, 1990). Based on this model, a hierarchical structured document should help facilitate information searching 
as it reduces time spent on category selection by trimming unlikely branches. 

However, with computers and the concept of hypertext, hierarchical and associative thinking can coexist, 
allowing learning of and searching through both organizational and associative relationships. It is generally agreed 
that hypertext systems should provide a proper organizational structure to enhance comprehension and infoimation 
retrieval (Herrstrom & Massey, 1989; Jaynes, 1989; Jonassen, 1988; Jonassen & Grabinger, 1990; McKnight, 
Dillon, & Richardson, 1991; Rubens, 1989; Simpson & McKnight, 1990), help form an accurate mental model of 
the systems (Holt, Boehm-Davis, & Schultz, 1986), and provide an "ideational scaffolding" (.uisubel, 1974) so that 
"a user's expectations can be confined and become more educative." (McAleese, 1989, p. 15). Some believe that by 
accelerating access without enhancing structural cues, it will be more difficult for users to make sense of where they 
are in the documents (Carey, Hunt, & Lopez-Suarez, 1990). In the paper world, a table of contents is usually given 
at the beginning to provide a systematic list of headings identifying the items discussed in the document; it is "an 
aid to both way-finding and sense-making" (Carey, Hunt, & Lopez-Suarez, 1990, p. 582). The power of 
hypertextual structure that builds on hierarchy is that it allows hierarchical and associative thinking to coexist. 
While tree-structured databases are often classified as non-linear, many systems often lack the kind of links that make 
a system a hypertext system: the referential links. Therefore, research is needed to find out whether and how users 
interact differently in strictly iuerarchically-structured systems and in hypertext systems that provide referential links 
in addition to hierarchical organization. 

Explicit Menus Vs* Embedded Menus 

In computer-based instruction, menus are "lesson-structuring devices.,, [The] design of a menu system 
imposes an implicit or tacit structure on the lesson which influences the learner's 'usage patterns'" (Schuerman & 
Peck, 1991, p. 93). One common type of menu is an enumerated list of possible choices, which explicitly signals 
the organizational structure. An alternative to such explicit menus are embedded menus: embedding the menu 
choices within the texts. Two possible drawbacks of explicit menus are verbosity and ambiguity, which are the 
results of extracting segments from the original context to form a list of menu items (Koved & Shneiderman, 1986). 
With embedded menus, on the contrary, it is easier to avoid ambiguity and unnecessary computer jargon or 
computer-related syntax and semantics issues (Carlson, 1989; Koved & Shneiderman, 1986). Context helps users 
better comprehend the menu items and make selection. 

This se!ection-ir. -context could prevent users from the so-called "Esc her Effect" (Ramey, 1989) when 
reading online text. From the paintings of M.C. Escher, viewers often experience a sense of disruption and 
dislocation (Jaynes, 1989). This sense comes from Escher's breaking the rules that govern the physical relationships 
between object* and the relationships between the creator and the thing created (Ramey, 1989). Explicit menus may 
be subject to such flaw because of their telegraphic style of menu items and lack of contextuality. Users of systems 
with embedded menus with rich contextuality may be less likely to experience such effect during reading. However, 
embedded menus have their problems too. Koved and Shneiderman (1986) propose three possible disadvantages of 
embedded menus. First, highlighting of phrases may be disruptive to readers, reducing reading speed and 
comprehension. Second, novice or inexperienced users may get lost more easily. Third, users may forget the 
original context in which the material was retrieved. Despite these suspicions, in their experiment of information 
searching in a database implemented in Hyperties describing the Student Union at a major university in the east, 
they found that embedded menus resulted in significantly better user satisfaction, fewer screens viewed, and more 
questions answered correctly (Koved & Shneiderman, 1986). The researchers concluded that all things considered, 
embedded menus seemed to be an attractive alternative for menu design. Their findings werf* contradictory to Rada's 
(1991) experiment in which Hyperties was compared with SuperBook, which has an explicit menu. 

While systems that do not provide clearly signaled structure, such as Hyperties, may satisfy a general 
audience's curiosity when browsing through an online museum exhibition information display (Shneiderman, 
Brethauer, Plaisant, & Potter, 1989), they may not meet the need of users who want to "find and do" (Herrstrom & 
Massey, 1989). The call for clearly signaled structure has grown stronger and researchers have come to realize that 
the provision of an appropriate organizational model will help users comprehend information better and facilitate the 



Lai & Waugh 



AERA '94 



4 



location and retrieval of specific information (Herrstrom & Massey, 1989; Jaynes, 1989; Jonassen, 1988; Jonassen 
& Grabinger, 1990; Rubens, 1989; Simpson & McKnight, 1990). Gluck (1989) points out that issues such as 
speed, graphical browser, user editing capability, and cost and time of development are "superficial" problems. 
These problems can be solved with increased computer speed and power. The real "deep" problems with hypertext 
are disorientation, cognitive overhead, and lack of presentation rhetoric. The first two deep problems have long been 
identified (Conklin, 1987). The third problem, according to Gluck, concerns the principles and rules of text 
composition and the ways the intended relationships among ideas are communicated to the learners. The strength of 
hypertext and important implications hypertext has for online instructional documents are its support for cross- 
reference and selection-in-context. Cross-referencing may give one faster access to associative information and allow 
one to see an information node from different perspectives. Selection in context may give one a greater sense of 
control and confidence in what one chooses to see. With clearly signaled structure and distinction of organizational 
and referential links, will the cross-referencing feature of hypertext systems still result in the problems of 
disorientation and cognitive overhead? Or will the intended relationships among ideas be more likely to be 
communicated? How well do the users interact with such systems? If both organizational and referential links are 
embedded within text, will this provide users with a more coherent presentation or is it the major source of 
disorientation? How well can users interact with such systems? Studies which address these issues have not yet 
been undertaken. 

RESEARCH QUESTIONS AND HYPOTHESES 

The questions raised in this study are: 

1. When search tasks are broken down based on their simplicity, specificity, and clarity of terminating 
conditions, will network structures facilitate certain type of search task? 

2. Will embedded menus facilitate information searching with certain types of search tasks? 

3. For search tasks that are complex, will network structures or embedded menus produce answers with 
greater wi Jth or greater depth? 1 

4. Which combination of document structures and menu designs produce better user satisfaction in terms of 
attitudes towards the document content and the program? 

5. Which combination of document structures and menu designs produce greater sense of getting lost and 
sense of digression? 

Method 

A document explaining the general computing and computer networking facilities and resources available 
on the campus of a major mid-western university was implemented into three different versions on Macintosh 
computers using the HyperCard program. The three sets of documents had the same organizational structure and the 
same user interface. Document Set 1 was purely hierarchical and used explicit menus. Set 2 and Set 3 were built on 
the same hierarchy as Set 1 but supported referential links additionally. Both Set 2 and Set 3 used embedded menus 
to signal referential links, but the hierarchical organization was signaled with explicit menus in Set 2 and embedded 
menus in Set 3. A summary of the differences among the three sets of documents is shown in Table 1. 



Table I 

Differences Among the T hree Sets of Documents 

Document Structure " Menu Design 



Set 



1 


hierarchy 




• explicit menus 


2 


network: 


hierarchical organization + 


• explicit menus for organizational links 






referential links 


• embedded menus for referential links 


3 


network: 


hierarchical organization + 


• embedded menus for both organizational links 






referential links 


and referential links 



^The width of an answer is defined as the number of search targets identified. The depth of an answer is defined as the 
number of details provided for each search target. 
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Figure I shows the main menu for Set 1 and Set 2 and Figure 2 shows the main menu for Set 3. The possible pros 
and cons of each set of document is given in Appendix A. Set 1 and Set 2 each consisted of 156 small units of 
information and 256 cards whereas Set 3 contained 147 units and 247 cards. This was a result of the different style 
of text presentation produced by Set 3's embedded menus which sometimes included information that was in the 
introductory sections in document Set 1 and Set 2. Information searching was decomposed into five categories based 
upon three criteria: specificity, complexity/focus, and boundary (see Table 2). The five categories were when the 
search tasks were (a) simple and fully known, (b) simple but partially known, (c) complex and fully known, (d) 
complex but partially known, and (e) complex and the condition for terminating the search was unclear. 
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O Introduction 

O Network Facilities and Resources 

O Password Protection 

O Getting Started on UNIX 

O E-mail 

O Other Facilities and Sendees on Mainframes 
O Major Networks and the UNIX System 
O Net Etiquette 



Figure 1 . Explicit menus used in Set 1 and Set 2. (Organizational selection items are listed explicitly.) 

Hypotheses 

As the real value oi hypertextual menu design may not be obvious unless the search tasks are complex or 
unclear, a trend was hypothesized: as search tasks became more complex or unclear, network structured document 
sets (Sets 2 and 3) would produce better search performance in terms of scores and time. A graphical representation 
of the hypothesized trend is shown in Figure 3. It was also hypothesized that the network structured document with 
explicit menus (Set 2) would (a) always result in the best score and fastest search and (b) be better received by users. 
The network structured document with embedded menus (Set 3) would (a) produce greater sense of getting Jost and 
sense of digression and (b) gradually result in better performance as search tasks become more difficult. Altogether 
six hypotheses were formulated: 

1. For searcn task 1 (simple and fully known), explicit menus (document sets 1 and 2) will produce higher 
search score and faster search than the embedded menu (document set 3). 

2. For search task 2 (simple but partially known), explicit menu and cross-reference (document set 2) will 
produce the best search score and the fastest search. Embedded menu (document set 3) will produce the worst search 
scores and the slowest search. 

3. For search tasks 3 and 4 (complex and fully known; complex but partially known), document set 2 will 
result in the highest search score and the fastest search. 

4. For search task 5 (complex and the condition for terminating the search is not clear), document sets with 
cross-references (document sets 2 and 3) will produce higher score and faster search. 

5. Document set 2 will produce the best user satisfaction towards the content and the program. 
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Subjects using the embedded menu (document set 3) will 

a. feel lost most often, and 

b. report getting distracted to side trails most often. 




This program describes shared computing services that 
are available throughout this campus via computer 
networks, known as UlUCnet* 

UlUCnet* is a backbone that allows access to networks 
such as Internet* , BITNET* and USENET* , You will 
learn about 

•) what network faciU ties and resources are available 



on 



tEe 



campus, 



•)how to ^protect ycmrpasswan^ 
•) how to ggt started on UNIX 
•) what &m^is and 

«)oiher facilities and services on mainframes. 

This program will also introduce you to the 
• i major networks and the UNIX syste m, and 
things to remember when you are involved in 
networking, or so-called, •) net etiquette, 




Figure 2 . Embedded menus usrd in Set 3. (Bolded, underlined words or phrases preceded by a •) mark indicate 
organizational selection items. Underlined words or phrases in plain text followed by an asterisk indicate referential 
links.) 



Table 2 

Breakdown of the Five Search Tasks 



Complexity 


Simple 


Complex 


Specificity/ 
Focus 


Fully Partially 
Known Known 


Fully 
Known 


Partially 
Known 


Boundary 


Known Unknown ; Known Unknown 
(NA) . (NA) 


Known Unknown 
(NA) 


Known Unknown 


Task# 


1 2 


3 


4 5 


Remarks 


Since there is only one search target in 


Since each search 


The search The search 




the search task, the terminating 
condition is always known. 


target in the task is 
fully known, the 
terminating 
condition is always 
known. 


targets are targets are 
not fully not fully 
known, but known, and 
the number the number 
of search of search 
targets targets 
sought is sought is 
clear. also unclear. 
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Task 




Score 
higher lower 


Time 

faster slower 






^ — P* 

S2>S3 


S2<S3 


1 




SI > S3 


SI < S3 


2 




S2 > SI > S3 


S2 < SI < S3 


3 




S2>S1 S2>S3 


S2< SI S2<S3 


4 




SI = S3 


SI = S3 


5 \ 




W S2 > SI S3 > SI 
f S2 R S3 


S2<S1 S3 < SI 
S2 = S3 



Figure 3 . Hypothesized trend for search performance. 

Note . The downward pointing arrow indicates the time factor as ail subjects proceeded from Task 1 to Task 5. 

All three document sets had three areas on each screen: the text area, the organizational hierarchy area, and 
the button control area (see Figure 4). The text area contained the actual information. It was also the area for menu 
selection. The organizational hierarchy area was located at the left hand side of each screen. It showed the ancestors 
of the current information node in the hierarchy. The ancestors were shown in a shadowed square button in bold- 
faced style, whereas the current node was shown in plain text style and an unshadowed square. A triangle pointer 
helped remind the subjects where they were in the hierarchy. Subjects could choose to go back to any ancestor by 
clicking on the shadowed squares. The button control area consisted of five buttons for document Set 1 and six 
buttons for Set 2 and Set 3. The "New Question" and "Stop" buttons were for the purposes of online data 
collection. Subjects were asked to click at these buttons to indicate the start and the end of searching for each 
question. The left-pointing and right-pointing arrows were for sequential reading, if so chosen. The "Help" button 
would branch the subjects to the help screen in which the organization and ways of navigation of the online 
documents were explained. Since Set 2 and Set 3 provided referential jumps, both had one additional "go-back" 
button to support backtracking. 

Evaluation Tasks 

For each search task and for each search question, several data items were collected during the experiment: 
(a) the length of time taken to find the answer, (b) the search path, which included each card visited and the time 
spent on that card, (c) the number of times subjects chose to see the help screen, (d) the number of times subjects 
utilized the left and right arrows for linear reading , and (e) for Set 2 and Set 3, as referential links were supported, 
the number of times subjects chose to branch through such links and the number of times they used the go-back 
button for backtracking. Search questions for each search task were selected by two independent reviewers from a 
question bank. Only questions that were classified into the same categories by both reviev/ers were adopted. Task 1 
and Task 2 each consisted of 4 questions, whereas Tasks 3, 4, and 5 each had 1 question (see Appendix B). A score 
for each search task was evaluated by the first author and two other raters. The width and depth of the answers to 
complex search tasks. Tasks 3 and 4, were also evaluated. As more than one component was involved in these 
complex search tasks, the width of the answers was defined as the number of components identified and the depth 
was defined as the details provided for each component. 

A subject attitude questionnaire consisting of thirteen items was administered at :he end of the experiment 
(Appendix C). The purpose of the questionnaire was to find out how subjects felt about the content and the program 
and to assess subjects' "sense of getting lost" and sense of digression during their search process. The questions for 
assessing the sense of getting lost were based on the two phenomena identified by Foss (1989) and the three sources 
of feeling lost according to Elm and Woods' (1985) study. Foss believes that users of hypertext systems often suffer 
the "Embedded Digression Problem" and the "Art Museum Phenomenon." The three sources of getting lost 
identified by Elm and Woods ( 1985) are: not knowing where to go next, knowing where to go but not knowing 
how to get there* and not knowing where they are in the overall structure of the document. 
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Internet 



The Internet is a worldwide collection of 
thousands of computer networks that can 
intercommunicate. All of them speak the same 
"language/ 1 namly the TCP/IP protocol suite. 
Users of any of the Internet networks can reach 
users on any of the other networks. The Internet 
started with the ARPANET, but now includes 
such networks as NSFNET, NEARNet, and 
others. Many other networks, such as BITNET, 
are tied to the Internet but are not an integral 
part of it. Approximately one million people use 
the Internet daily. 
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Text Area 



Button Control Area 



Organizational 
Hierarchy Area 

Figure 4 . Screen layout for the three sets of on-line documents. (The go-back arrow at the bottom right-hand side 
of the screen is not supported in Set 1 except in cases where subjects branch to the help screen and maj.»s.) 



Procedures 

The experiment was carried out on Macintosh SE computers in a computer lab. Subjects participated in the 
experiment two at a time. This was an effort to ensure all subjects clicked at the "New Question" and the "Stop" 
buttons at the right time. Subjects first spent about 10 minutes viewing an introductory program that explained the 
procedure of the experiment, the organization of the online document, the function of each button, and the ways to 
navigate. Then they were shown the actual online document they were going to use and had five minutes to get 
familiar with navigation and the function of each button. Each subject then carried out all five search tasks, from 
search task 1 to search task 5. The sequence of search tasks was fixed for all subjects, progressing from the simplest 
to the most difficult. For the eight questions in Task 1 and Task 2, they had a maximum of ten minutes for each 
question. For Tasks 3, 4, and 5, they had a maximum of fifteen minutes to search for answers and take notes. After 
the fifteen minutes, they had all the time they needed to write down the answer, but were advised to write down 
related concepts and information in a structured way and not to bother writing complete sentences. Finally they 
filled out the attitude questionnaire (Appendix C). 

Data Analysis 

While comparisons of time spent on searching and accuracy of search can reveal the efficiency of the three 
interfaces, "such measures... have less obvious relevance to the process of learning" (McKnight, Richardson, & 
Dillon, 1990, p. 288). In order to better understand how the different document structures and menu designs affected 
the way the subjects navigated, search strategies utilized for search tasks 2, 3, and 4 were analyzed based on the 
sequence of visits to cards. These three tasks were chosen to be analyzed because they took the most time to 
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complete among the five and significant differences in search scores were found in these tasks. The analysis was 
based on a scheme characterizing search strategies proposed by Canter, Rivers, and Storrs (1985). 

Canter, Rivers, and Storrs draw on three lines of research and propose a detailed characterization of 
information searching strategies. The first line of research recognizes the parallels between navigation in databases 
and physical navigation in concrete environments. The second line of research studies social and psychological 
implications of navigation by providing algebraic specification of the physical structures. The third line of research 
brings in path algebras for specification and study of interactive systems and database navigation. The five different 
search strategies they propose are: scanning, browsing, searching, exploring, and wandering, and characterization of 
these strategies is based on six indices: pathiness, ringiness, loopiness, spikiness, number of unique nodes visited 
vs. number of nodes visited (NV/NS), and number of unique nodes visited vs. total number of nodes (NV/NT). See 
Appendix D for a definition of the six indices and the characteristics of the five search strategies. 

As a program that will calculate the paths, rings, loops and spikes automatically and correctly from the data 
has not yet been written, data for five subjects from each treatment group were selected for manual analysis to find 
out if referential links or different menu designs encourage certain search strategies. Selection was made by listing 
the subjects in each group ascendantly according to their total scores and the fourth, eighth, twelfth, sixteenth, and 
twentieth subjects were chosen. The search paths were plotted on a sheet containing the hierarchical structure of the 
document sets for calculation of pathiness, ringiness, loopiness, and spikiness. They also were loaded into 
FoxBASE for calculation of NV and NS. 

RESULTS 

Sixty-nine undergraduate students from the subject pool of the Department of Educational Psychology at a 
major university in the midwest volunteered to participate in the study as part of their course requirement. They 
were assigned to three treatment groups according to \he sequence they came in for the experiment and thus resulted 
in a balanced design with 23 subjects in each group. The sample consisted of 17 males (24.64%) and 52 females 
(75.36%). Their ages ranged from 18 to 23 years old, with 2 freshmen (2.90%), 27 sophomores (39.13%), 23 
juniors (33.33%), and 17 seniors (24.64%). The majority of the subjects were Education majors (42 subjects, 
60.87%) while Liberal Arts and Sciences majors took up 2 1 .74% (15 subjects), Agriculture majors, 8.69% (6 
subjects), Fine and Applied Arts, 4.35 % (3 subjects), and Applied Life Studies, Commerce and Business 
Administration, and Communication each with 1.45% (1 subject). 

Hypotheses Testing 

Hypotheses 1-4: 

The results of the first four hypotheses testing are shown in Table 3: 

Table 3 

Results of First Four H ypotheses Testing Concerning Score and Time 



Taak 


Score 

higer lower F p 


Time 

faster slower F p 


1 


S2 = S3 = SI 


2.27 0.1109 


SI = S2 = S3 


0.10 0.9090 


2 


S2>S3 


4.44 0.0155 


SI = S2 = S3 


2.85 0.0649 


3 


S3 = S2 = SI 


1.99 0.1444 


S2 = S3 = SI 


0.99 0.3785 


4 


S2 > SI S3 > SI 


4.13 0.0204 


SI = S2 = S3 


1.95 0.1505 


5 


S3 - SI = S2 


2.03 0.1398 


S2 = SI = S3 


0.41 0.6658 



Hypothesis 5: 

Set 2 will produce the best user satisfaction towards the content and the program. 

No difference was found for subjects' attitude toward the content (F=1.45, /?=0.2413). However, Set 2 was 
significantly more-popular than document Set 3 and Set 1 (F=7.18 t p=0.0015). 
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Hypothesis 6: 

Subjects using the embedded menu (Set 3) will 

a. feel lost most often, and 

b. report getting distracted to side trails most often. 

Both failed to show any significant difference among the three groups of subjects (sense of getting lost, 
F=0.15, p>=0.8588; sense of digression, F=1.44, p=0.2445). 

Further Analysis 

As reported earlier, for Task 3 and Task 4, search answers were rated in two categories: width and depth. 
The previous hypotheses testing was based on the sum of the two categories. Significant differences were found in 
the width of answers for Task 3 ^F=3.97, /7=0.0235) and both Set 2 and Set 3 produced higher scores in width than 
Group 1 (a=0.05) but not in depth (F=0.33, p>=0.7236). For Task 4, significant differences were found in the width 
of answers (F=3.33, p=0.0417) as well as in the depth (F=3.32, /7=0.0424). Set 2 produced significantly higher 
scores in width than Group 1 (a=0.05) while Set 3 produced significantly higher scores in depth than Set 1 (a=0.05). 
Therefore, when the subjects were not sure about what and how many components they were looking for, network- 
structured document sets produced better answers in both width and depth, with the explicit menu (Set 2) facilitating 
the width of search and the embedded menu (Set 3), the depth of search. As for search time, no significant 
differences were found among the three groups in the time they spent on each of the five search tasks. Even though 
Set 3 seemed to cost a little more time for most of the tasks, an analysis of variance of the total time spent on all 
search tasks did not show significant differences (F=2.75,/7=0.071 1). Based upon the findings, the trend 
hypothesized in Figure 3 needs to be modified and is shown in Figure 5. Note that in the following discussion, 
Group 1 refers to subjects using document set 1; Group 2, Set 2; and Group 3, Set 3. 



Scores 



Task 



The Whole Task 



S2 = S1 =S3 



S2>S3 



S3 = S2 = S1 



S2 > SI S3 > SI 



S ^ S3 = S1=:S2 
▼ 



Width 



S2>S1 S3 > SI 



Depth 



S3 = S2 = S1 



S2>S1 



S3 > SI 



I 



Figure 5 . Modified graph of overall trend for search accuracy. 

Note . The downward pointing arrow indicates the time factor as all subjects proceeded from Task 1 to Task 5. 
Other Variables 

Five other variables were further analyzed: (a) number of times referential links were utilized, (b) number 
of times go-back links were utilized, (c) number of times forward arrows were used, (d) number of times backward 
arrows were used, and (e) the total number of visits to cards. For the use of referential links, Group 3 was found to 
use referential links significantly more often than Group 2 in Task 2 (t— 5.98, p=0.0185) and Task 3 (t=7.99, 
/?=0.0070). Overall, Group 3 used the referential links more often than Group 2 (t=7.61,p=0.0084). As for use of 
go-back links, even though no significant difference was found in individual tasks, overall speaking, Gr^ip 3 
utilized go-back links significantly more often than Group 2 (F=4.56, p=0.0384). No significant difference was 
found in either the use of forward or backward arrows, and no significant difference was found in the total number of 
visits to cards. 
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Correlation 



Correlation among all the previously analyzed variables was examined. Two clear patterns can be seen in 
these tables: 

1 . Time spent on searching was positively correlated with the number of visits to cards for all f;"* search 

tasks. 

2. In most cases, the use of forward arrows were positively correlated with the use of backward arrows, and 
the use of either forward or backward arrows were positively correlated with the number of visits to cards. 

Several interesting findings are: 

1 . Only in Group I did attitude towards the program and the sense of getting lost correlate with total score. 

2. A positive correlation is found between the use of referential links and the sense of getting lost in 
Group 2 while no such correlation is found in Group 3. 

3. In Group 2 the use of go-back links is positively correlated with the sense of getting lost and negatively 
correlated with the attitude towards the program, while in Group 3 the use of go-back links did not contribute to 
greater sense of getting lost or more negative attitude towards the program. 

Search Strategies 

The strategy utilized by Group 1 can be classified as "scanning" (high NV/NT, high spikin^ss, medium 
loopiness), and strategy utilized by Group 2 is "exploring" (high pathiness and high NV/NT).^ However, no search 
strategy seems to fit the values found for Group 3 really well. Its ringiness is the highest among the three groups, 
but its NV/NS is not low enough and its NV/NT does not seem high enough to classify the search strategy as 
"wandering." With medium loopiness and medium NV/NS, "browsing" seems a better fit than the other four 
strategies. 

DISCUSSION 

According to the findings, the interface used by Group 2 produced significantly better search scores in Tasks 
2 and 4, i.e., the network-structured document set with explicit menus for organizational links promoted search 
accuracy for tasks that were only partially known regardless of their complexity. The interface used by Group 3 
produced significantly better search scores in Task 4, but significantly worse scores in Task 2. When each search 
component in a complex search task was known (Task 3), both interfaces used by Group 2 and Group 3 only helped 
the subjects identify the components in the search task and did not result in more detailed answers for each 
component. When the search components in a complex search task were not fully known (Task 4), the interface Set 
2 employed helped the subjects identify the components while the interface Set 3 employed encouraged greater details 
for each component. This superiority of network-structured documents would not have been detected if a breakdown 
of search tasks was not carried out and al! search questions had been simple and fully known. Equally important is 
the finding that embedded menus are good for complex search tasks (Tasks 3 and 4), yet not ideal for tasks that are 
simple and not fully known. These findings echo the calls in the literature for an increase of the specificity of 
learning tasks (Dillon, 1990) and a decomposition of search tasks (Wright, 1990) in related research. These findings 
may also help explain why earlier studies on information searching in hypertext environments with various menu 
designs have conflicting results. 

As discussed earlier, information searchers are often goal-oriented and have little motivation to go beyond 
what is asked of them. This experiment shows that when what to look for is clear, network-structured documents do 
help information searchers identify the search targets, but they may not provide more motivation for information 
searchers to go beyond what is required, especially in this experimental setting where the subjects participated 
because of a course requirement. Only when what to look for in a complex situation is unclear will the network- 
structured document with both organizational and referential linkc signaled by embedded menus (Set 3) promote in- 
depth searching. The reason why Set 3 encouraged in-depth search in Task 4 is hard to identify. Comparison of the 
use of referential and go-back links, NV/NT, and NV/NS for Task 4 all failed to show any significant differences 
with Set 2 or Set 1. There are three possible explanations for this superiority of Set 3 ti\ Task 4. First, while 
Group 3 did not view more cards or use more referential links, the cards they viewed might have been very different 
from what Group 1 and Group 2 viewed or they might have viewed the cards in different sequences or different 
contexts. Second, in general, Group 3 used more referential and go-back links than Group 2. By the time they 

2 Categorization of search strategies for the three groups is based on visual inspection of search patterns used by 15 
subjects 
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worked on Task 4, they might already have had a different perspective or mental model for the document. This issue 
of mental models will be further discussed in a later section . Third, subjects in Group 3 who used more go-back 
links had stronger desire to digress, and Task 4, with its complexity and vagueness, is the best candidate among the 
five to encourage any such desire. 

The result that search scores across groups for Task 5 were not significantly different means that when the 
size of the document was small enough for exhaustive searching within one sitting and after the subjects had 
searched the document for other various search tasks, search performance of the three groups for this type of search 
task was not significantly different. That is, all groups developed a similar "big picture" of the content domain after 
going through Tasks 1, 2, 3, and 4. After sitting and reading from the screen for more than an hour, neither group 
was more willing or able to provide greater details. Since in this study the search tasks were sequenced from the 
simpler to the more complex, this type of sequence as well as the time factor involved could confound the findings. 
Separate studies of each search task may yield different resulrs. The fact that the three groups of subjects only spenr 
an average of 4.23 (Group 2) to 4.92 (Group 3) minutes for search Task 5 showed that the subjects simply went 
through the online document in an attempt to "wr^p-up" what they had read. 

Attitudes 

As hypothesized, subjects in Group 2 showed significantly more positive attitudes toward the program they 
used as compared with Group 3. However, with the other three categories, the three groups did not differ 
significantly. That is, the structure of the document (hierarchy vs. network) or the design of the menu (explicit vs. 
embedded) did not seem to affect the subjects' attitudes toward the content, nor did they seem to cause significant 
difference in the subjects' sense of getting lost or their tendency to digress. This might be because the size of the 
document was small enough for the subjects to become familiar with the overall structure in two hours, especially 
when all three document sets were based on the same hierarchy. As the focus of this study is on small or medium 
sized online documents, it is important to mention that the same result may not be applied to large documents. 

Correlation 

Findings concerning the use of referential and go-back links are very surprising and intriguing. Since the 
only difference between Set 1 and Set 2 was the availability of referential and go-back links in Set 2, the 
discrepancies in search scores should be a result of using these links. However, the use of referential and go-back 
links was not correlated with search scores yet was positively correlated with the sense of getting lost and negatively 
correlated with attitudes toward the program for Group 2. For Group 3, the use of either type of link was not 
correlated with scores or the sense of getting lost, yet the use of go-back links was correlated with the sense of 
digression. 

Similarly perplexing are the findings that even though Group 2 liked their program better, attitude towards 
the program was not correlated with search accuracy or efficiency. Neither was it correlated with the use of 
referential links. Actually, those in Group 2 who used more go-back links had a less favorable attitude for the 
program (r=-.43, p<0.05). Moreover, the use of referential links was not correlated with search scores for Group 2 
and Group 3, and the use of referential links and go-back links was positively correlated with the sense of getting 
lost for Group 2. As a matter of fact, all these findings ruled out the initial conjectures that if Group 2 and Group 3 
had better search performance than Group 1, use of referential links must be the cause. Table 4 lists these initial 
conjectures and the actual findings regarding attitudes and the use of referential and go-back links. 

With the results of the study failing to support the initial conjectures, there must be other less obvious 
reasons for these discrepancies in search performance and attitude towards the program. For the correlation between 
search scores and attitude. Group 1 is the only group that showed a positive correlation between search scores and 
attitude towards the program and a negative correlation between scores and sense of getting lost. These correlations 
seem intuitively natural, but such relations did not exist for Group 2 or Group 3. This may be an important 
indication of how referential links change the usual way information searchers use online documents and these 
changes may trigger mixed feelings in the users. This ambivalence, if it exists, unfortunately, could not be detected 
by the questionnaire designed for this study. 

Two possible explanations for the better search performance by Group 2 and Group 3 are (a) aesthetic 
appeal and (b) major concepts/terminology hinting. Aesthetically, Set 2 might be more appealing to the eyes than 
Set 1 as it provided some variation in the style of the text, e.g., underlining. Set 3, even though it also provided 
variation in the style of the text, might have been too eye-dazzling and complex for its users as both organizational 
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Table 4 

Initial Hypotheses and Actual Findings Concerning the Difference in Attitude and Use o f Referential and Go-Back 
Links 



Correlation 

Related 



Variables Initial Conjectures Actual Findings 



• Attitude towards the Would be POSITIVEL 
program for all groups 

• Scores 


,Y correlated Were POSITIVELY correlated only in 
Group 1 


• Sense of getting lost Would be NEGATIVE 

• Scores for all groups 


LY correlated Were NEGATIVELY correlated only 
in Group 1 


• Use of referential links Would be POSITIVEL 

• Scores for both Group 2 and i 


,Y correlated Were NOT correlated in either group 
3rouo 3 


• Use of go-back links Would be POSITIVEL 

• Scores for both Group 2 and < 


,Y correlated Were correlated in some tasks for 
Group 3 Group 2 


• Use of referential/go-back Both links would be P 
iluk3 correlated with attitude 

• Attitude towards the program for Group 2 a 
program 

• 


OSmVELY Use of referential links was NOT 
; towards the correlated with attitude towards the 
ind Group 3 program for either group 

Use of go-back links was 
NEGATIVELY correlated with 
attitude towards the program for 
Group 2 



• Use of referential/go-back Both links would be POSITIVELY Both were POSITIVELY correlated 
links correlated with sense of getting lost with sense of getting lost for Group 

* Sense of getting lost for Group 3, not Group 2 2, not Group 3 



and referential links were signaled inside the text. Set 2 was probably the one that resembled paper texts the most in 
that hierarchical organization was explicitly represented, similar to the hierarchical breakdown of chapters, sections, 
and paragraphs, and that referential links were signaled within the text, similar to the "see also" convention in paper 
texts. These might explain why users of Set 2 indicated stronger affinity for this interface. Secondly, referential 
links in Set 2 and Set 3 might provide visual hints to the subjects by repeatedly holding or underlining the major 
concepts and terms in the documents, which is another convention in the paper world and is something readers and 
information searchers learn from their experience with paper texts. As a result, the subjects might be more willing 
to try out these "emphasized" phrases. 

However, subjects using Set 2 might make less mental effort to keep track of their places and thus have a 
fuzzier or less appropriate "mental model" of the "information space" as compared with Group 3. A "mental model" 
is a user's mental image of the structure and internal relationships of the system, and it helps the user to not only 
understand the system but also draw inferences and make predictions about the system's behavior (Borgman, 1986; 
Norman, 1983; Manktelow & Jones, 1987; Rupietta, 1990). It consists of both structural and procedural knowledge 
about the system (Rupietta, 1990). An appropriate mental model can help the user better cope with the system, but 
an inappropriate one can lead to errors (Young, 1981). Using explicit menus to signal hierarchical structure is a 
common practice in electronic documents whereas signaling both hierarchical and referential relationships through 
embedded menus is a newer practice. Therefore subjects in Group 2 might have had lower anxiety (see Appendix A 
for a list of possible pros and cons of Set 2) and thus made less mental effort to identify where they were in the 
document and the relation between where they were and where they came from. In short, they developed a fuzzier 
mental model. Group 3 often used go-back links right after their use of referential links, possibly in an attempt to 
figure out where they were. Frequent use of referential links by Group 2 without constant effort to keep track of 
places may have resulted in a sense of disorientation. Subjects in Group 3 seemed to make more efforts to keep 
track of their places in the information space by using go-back links right after the use of referential links and the 
data show they went back and forth several times, as indicated by more small loops and rings in Group 3's search 
paths. 

As mentioned earlier , Bolter (1987) argues that going back and forth and reviewing nodes several times 
may help learners synthesize what they read. It is possible that the unfamiliar interface presented by Set 3 initially 
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increased Group 3's anxiety and sense of disorientation and therefore their effort to find out the connection among the 
nodes. Anxiety and disorientation can be a good thing for learning (Mayes, Kibby, & Anderson, 1990). This 
going-back-and-forth phenomenon might be a strategy Group 3 employed to find out where they were and the 
relation between the n' des connected through referential links. As the experiment went on, their sense of 
disorientation might have gradually diminished so that the questionnaire administered at the end did not detect a 
greater sense of getting lost for this group. Group 2, on the other hand, might not have paid as much attention as 
Group 3 to the organization and connection of the information nodes, and, therefore, those who used more referential 
or go-back links possessed a greater sense of having felt lost. 

It may be too naive to expect that the differences in search scores between Set 1 and the other two network- 
structured document sets was caused by the u:e of referential links and that there was a perfect relationship between 
the use of referential links and search scores. It is possible that the existence of the referential links made a 
difference. It is also possible that some use of such links may help but careless use or overuse may have a negative 
effect on search performance. Knowledge of the organization of the information space may play an important role if 
information searchers are to take advantage of network branching. What is interesting is that the use of referential 
and go-back links was positively correlated with the sense of getting lost for Group 2 but not for Group 3. This 
could mean that either embedded menus did provide a greater sense of context or embedded menus "forced" the 
subjects to use referential links and go-back links together more often. Note that the use of go-back links was 
positively correlated with the sense of digression for Group 3. showing that network-structured documents may 
better encourage pursuit of interest if the menus are embedded. 

Actually both Group 2 and Group 3 had trouble using go-back links in the way they were designed tc be 
used-right after the use of referential links. While there is nothing wrong with using the go-back links as a quick- 
and-dirty way to review cards that were most recently visited, such use might lead to greater sense of getting lost or 
might be an indicator of the subjects' disorientation in the information space. This is especially obvious with 
Group 2, whose use of referential and go-back links was correlated with their sense of getting lost. Table 5 shows 
the correlation between the use of go-back link* and the use of referential links. The two did not become 
significantly positively correlated until Task 3, indicating lhax perhaps the subjects needed some time to get familiar 
with such links. Table 6 shows the correlation between the use of go-back links and search scores. Notice that for 
Group 2, the correlation progressed from very negative to positive to significantly positive, whereas no such trend 
existed for Group 3. This shows that Group 2 initially had greater difficulties in using go-back links than Group 3, 
which was possibly the cau!»e of their greater sense of getting lost. 



Table 5 

Correlation Between Go-Back Links and Referential Links 



Group 


Task 




1 


2 


3 




4 


5 


Overall 


2 




.08 


-.20 


.51* 




.34 


.79** 


.19 


3 




-.02 


.12 


.48* 




.33 


.60** 


.25 


*Z <0.05. 


**£ < 0.01. 














Table 6 


















Con-elation Between Go-Back Links and Search Scores 




























Group 


Task 




1 


2 


3 




4 


5 


Overall 


2 




-.47* 


-.28 


-.15 




.04 


.48* 


.16 


3 




-.25 


-.07 


-.15 




-.24 


.16 


.17 



*2 <0.05. 
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Search Strategies 

Based upon Canter, Rivers, and Starrs' search strategy classification, users of Set 1 "try to cover a large area 
without deep depth," users of Set 2 "are seeking the extent and nature of the field," and users of Set 3 "are happy to 
go wherever the data takes them until their interest is caught." Since document set 1 only supported organizational 
links, its users were more likely to be "forced" to retrace the in-coming path in order to get out or to branch to 
another section. This limitation on branching resulted in more spikes in the search path and encouraged the 
"scanning" strategy. Set 2, on the contrary, encouraged its users to "seek the extent and nature of the field" b^ 
providing referential links, which not only offered visual hints of important terms and concepts but also allowed 
easy access to those sections. 

It is interesting that no good fit is found for Group 3, and yet describing them as "happy to go wherever the 
data took them until their interest was caught" seemed to be an accurate description for those who used more go-back 
links and had a greater desire to digress (r=.47, /?<0.0I). As this desire to digress was not significantly correlated 
with search scores (r=.39), it could mean that sometimes the subjects did tend to follow wherever the links led them 
and therefore did not have better search scores in some search tasks. It also could mean that Set 3 did present a new 
combination of document structure and menu design that can not be perfectly described and categorized by this 
scheme. As a matter of fact, the assumption that high ringiness means wandering or high loopiness means 
mindlessness might need further investigation. In Group 3, there were many small loops and rings that involved 
only two, three, or four cards. Many of these happened when the subjects used the referential links and go-back 
links together back and forth, possibly in an attempt to find out where they were. 

Canter, Rivers, and Starrs' scheme does provide a good start and a useful tool for analysis of information 
searching activity in hypertextual environments (McAleese, 1989), but this analysis identified three problems with 
this classification scheme. First, in their comparison of search strategies promoted by menu- vs. command-driven 
interfaces (Canter, Rivers, & Starrs, 1985), the network database only supported uni-directional links. As a result, 
there would not be the type of small foops or rings as were produced by Group 3 in this experiment. Second, this 
scheme does not take into consideration the length of loops, paths, rings, and spikes, which may provide important 
information about what really happens. Third, calculation of the loops, paths, rings, and spikes as defined is a 
difficult task. Canter, Rivers, and Starrs (1985) also realized this problem: 

[The] indices do present some calculation problems and should not yet be considered a finished tool for 
characterizing user navigation. Problems of overlap of some of the indices and the need to develop more 
complex and subtle algorithms remain to be solved. This said, the indices do seem to have present 
potential as a means of improving the understanding of search strategies, while their continuing 
development should further improve their power in this respect, (p. 99) 

Since their scheme was developed in the mid 80's before hypertext became a popular concept, some modifications 
may be necessary for this scheme to better describe information searching activity in hypertextual databases. 

CONCLUSIONS 

While it is impossible to fully understand the potential of hypertext by studying small- or medium-sized 
electronic documents, providing network structures and referential links for documents of such sizes can still affect 
search performance and users' attitude. The following conclusions can be drawn from this study. For easier 
description and understanding, the term EXH (explicit and hierarchical) is used for the kind of document structure and 
menu design employed in Set 1, EXN (explicit and network) in Set 2, and EMN (embedded and network) in Set 3. 

1. This study showed that network-structured documents could improve search accuracy though not 
efficiency . In general, EXN worked best. EMN worked best when in-depth search for questions that were complex 
and only partially known was desired. Neither document structures nor menu designs significantly affected the time 
spent on searching. The results imply the following: 

a. In an information search task in which the search target is simple yet unclear, EXN can produce 
better answers than EMN. 

b. In an information search task in which multiple search targets are involved yet each search 
target is clear, network-structured documents (both EXN and EMN) can produce better answers in width, 
regardless of menu designs. 
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c. In an information search task in which multiple search targets are involved yet not all search 
targets are clear, network-structured documents can produce better answers in both width and depth, with the 
explicit menu (EXN) facilitating the width of search and the embedded menu (EMN), the depth of search. 

d. In general, for medium-sized information spaces, information searchers will develop a similar 
n big picture" of the document after searching for various tasks regardless of the document structure 
(hierarchy or network) or menu design (explicit or embedded). 

2. Information searchers favored the EXN design . This was likely caused by the visual appeal of various 
text styles or the freedom to branch to another topic of interest. It also could be a result of the similarity of such 
design to paper texts. EMN did not appeal to its users as much. This could be caused by the cognitive burden it put 
on the users, who needed to be more conscious of what they clicked and where they went as a result of the click. 

3. EMN encouraged use of referential links and back t racking after such links without resulting in greater 
sense of getting lost . Those who used go-back links more often also showed a stronger desire to digress. Therefore, 
the kind of interface EMN presents can encourage greater mental effort to keep track of places in the information 
space than EXN can. 

4. Users of EXN who used more referential or go-back links showed greater sense of getting lost . The 
familiar interface and quick cross-referencing capability might cause EXN users to be less aware of their location in 
the information space as compared to those EMN users. 

5. The, search strategy utilized bv EXH us ers appears to be "scanning;" EXN. "exploring;" and EMN, 
closer to "browsing." However, this categorization does not provide a complete picture of what goes on in the 
searchers' mind. 

6. Different combinations of document structures and menu design did not produce significant differences in 
the ratio of number of inte.- ; or nodes visited vs. number of leaf nodes visited, the ratio of time spent on viewing 
interior nodes vs. time spent on viewing leaf nodes, the ratio of unique nodes visited vs. total number of nodes, or 
the ratio of unique nodes visited vs. number of visits to nodes. What may be different are the nodes visited, the 
sequence of the visits to nodes, and how the information searchers perceive the association of the information nodes. 

7. It takes time for users of EXN and EMN to become famil iar with the use of referential links and 
backtracking after such use . How such use affects users' attitudes is unclear. 

RECOMMENDATIONS FOR FUTURE RESEARCH 

Future research should be conducted in the following four areas: (a) a well-defined algorithm and a more 
detailed scheme for search strategies categorization, (b) qualitative studies on search behavior, (c) modification of this 
study, and (d) similar studies on large documents. 

Future research on search strategies can benefit from a more detailed scheme and analysis of search paths by 
taking into account the length and size of loops, paths, rings and spikes, which is not considered in Canter, Rivers, 
and Storrs' scheme. If a certain document structure or menu design promotes searching or exploring, it may not 
increase the number of paths but rather increase the length of paths. A more detailed definition for these four indices 
is also critical for future analysis as counting and tallying by hand can be subjective and error-prone. Once a well- 
defined algorithm is available, a computer program that can automatically calculate the number of occurrences and 
the size of each occurrence for the four indices can be written to help analyze data on a larger scale. 

Even when we know how many loops and rings there are in one's search path, it is still dangerous to 
assume that loopiness and ringiness necessarily means getting lost or "mindless wandering." In this experiment. 
Group 3 had more rings and larger loops, but subjects in this group did not report feeling lost and their search 
performance certainly confirmed this. Canter, Rivers, and Storrs' scheme, while valid for their study with a uni- 
directional network-structured database, may need some modifications for study of bi-directional or multi-directional 
network structures like hypertexts. It is also difficult to find out what is in the users' mind unless some qualitative 
measures are made, such as videotaping during the experiment and interviews afterwards. Future studies should 
include these types of measures. 

Several modifications could be made to improve replications of this study. Future research should design 
separate studies on each search task, especially with tasks whose terminating condition for search is unclear. Unlike 
this study which let search experience build on earlier search tasks, separate studies may show different results. 
Also, data collection may include the number of times go-back links are used right after referential links. This piece 
of information may provide insight in the actual use of go-back links and help explain why such links cause 
confusion for some but not for others. Furthermore, the attitude questionnaire should be modified to investigate 
how the searchers feel about using referential and go-back links and how such use might influence their sense of 
getting lost and desire to digress. A formative rather than summative evaluation of the searchers 1 anxiety, sense of 
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disorientation and sense of digression would provide greater insight in how information searchers feel, what affects 
their attitudes, and how their attitudes change in the course of time. 

Similar studies should be conducted on large documents or sets of documents that cannot be exhaustively 
searched or browsed within one sitting. The larger the document size, the more easily one can get lost in the 
information space. Thus, chances are a difference in search performance and search patterns is more likely lo be 
found. However, as the document size increases, categorization of search tasks may change. The word "boundary" 
may mean the extent to which a search component can be explored rather thun the number of search components in a 
task. Moreover, as the documents become more complex, associative links may get more complicated than just 
referential. Typed links may need to be provided in order to fully represent this richness of connectivity. But no 
matter how the search tasks or associative links are categorized, future studies should investigate the searchers' 
mental model of the information space, the cause of differences in their mental effort to keep track of places, and 
how a system or interface design can promote greater mental effort and an appropriate mental model without 
endangering the searchers' confidence in their own control of the search process. 
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APPENDIX A 

POSSIBLE PROS AND CONS OF EXPLICIT AND EMBEDDED MENUS 



Set I. Explicit Menu WITHOUT Cross-References 



Advantages 



Disadvantages 



• Structural organization is clearly signaled. 

• Searching for specific, well defined facts may be 

easier. 

• Learners are familiar with such menu design. 

Therefore, users may have low anxiety in using 
the program. 



1 Names for menu items may be arbitrary and therefore 
less well understood by users. 

► Searching for less well defined purposes may be 

difficult. 

► Learners may not have much chance to see the 

associative relationships among nodes. 

► Accessing information in other branches of the 

hierarchy will be difficult. 



Set 2. Explicit Menu WITH Cross-References 



Advantages 



Disadvantages 



• Structural organization is clearly signaled. 

• Searching for specific, well-defined facts may be 

easier than embedded menu (Set 3). 

• Searching for less well-defined questions may be 

easier than explicit menus without cross- 
referencing capability (Set 1 ). 

• Quick access to associative information nodes is 

made easy. 

• Learners may better see the associative 

relationships among information nodes. 

• Learners may have lower anxiety in using such 

programs, compared with those using embedded 
menu. 

• Learners may have the lowest anxiety in learning 

the conteht. 



► Names for menu items may be arbitrary and therefore 

less well understood by users. 
» Searching for less well defined purposes may be 
difficult. 

» Learners can be more easily distracted by side trails 
than those using strictly hierarchical documentation 
and forget about the original purpose of searching or 
reading. 

► It takes time for learners to get used to using embedded 

links. 



Set 3. Embedded Menu for BOTH Structural Organization and Cross-References 



Advantages 



Disadvantages 



• Quick access to associative information nodes is 

made easy. 

• Learners may better see the associative relationships 

among information nodes. 

• More information is provided from which learners 

can judge whether to follow a certain branch or 
side trail or not. 



Learners may not be familiar with such menu design. 
■ Learners have to be able to distinguish between 

organizational and referential links in order to use 

such systems comfortably. 

► It may take longer time for learners to learn to use 

such menus as compared to explicit menus. 

> Learners may be easily distracted by side trails and 

forget about the original purpose of searching or 
reading. 

1 Learners may not be able to form a coherent 
organization for what they have seen. 
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APPENDIX B 
SEARCH QUESTIONS 



Task I. Simple and Fully Known 

1 . What command should I use on UNIX to change my password? 

2. What UNIX command can I use to find out the e-mail address of my friends on this campus? 

3. When does the computer lab in Illini Union close on Friday night? 

4. What session name should I enter when using Telnet to connect to the Illinet Online Plus (10 Plus), the new 
online library catalog system? 

Task 2. Simple but Partially Known 

1 . Is there help available on UNIX that explains how to carry out an online conversation with another person on 
uxa? How can I access such information? 

2. What command should I use to see what files I have and when they were last modified in my uxa account? 

3. I was playing around on uxa and accidentally got into a program. I couldn't get out of the program since there 
is no instruction on the screen telling me how to quit, so I just turned the terminal off. Is that OK? What 
should I do in case similar things happen again? 

4. My friend gave me a file GBuster.cpt that she got from the mainframe. She said it is a free program for 
Macintosh computers. But how come when I tried to open it, it said something like "The file cannot be 
opened. The application is either busy or missing?" 

Task 3. Complex and Fully Known 

I. I've heard about (1) Internet, (2) BITNET, (3) Telnet, (4) UNIX and (5) uxa. But what are they and 
how are they related? 



Task 4. Complex but Partiaily Known 

1 . I have an IBM compatible computer at home. What else do I need in order to get on to the University's 
online library catalog > searching, checking out, and renewing books from home? Do I need to have a 
mainframe account? Does it make any difference if I don't? 



Task 5. Complex and Condition for Terminating Search is Unclear 

1 . I was told that as a U of I student I can get a free UNIX account. What can I do with this UNIX account? 
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APPENDIX C 
ATTITUDE QUESTIONNAIRE 



Scal es 



Questions 



I. 
2. 
3. 
4. 



Attitudes Towards the Content 
Attitudes Towards the Program 
Sense of Getting Lost 
Desire to Digress 



2, 6, 7 

I, 3, 8, 12, 13 
4, 5, 9, 10 
11 



The following statements are intended to find out what you felt about the online document that you just read. Please 
note that throughout the questionnaire "the program" or "this program" is used to indicate the online document. For 
each statement, please circle the number that corresponds to the degree to which you agree or disagree. 

Strongly Disagree < > Strongly Agree 

2 3 4 5 6 7 

2 3 4 5 6 7 

2 3 4 5 6 7 

2 3 4 5 6 7 

2 3 4 5 6 7 



1 . I felt that I could find answers to the questions quite easily. 

2. I found the content of this program to be interesting. 

3 . The program was easy to use. 

4. I often had trouble figuring out where I was in the program. 

5. I often had trouble finding a section I know was there 
somewhere. 

6. I feel more comfortable about the subject of networking 
facilities on this campus than I did before. 

7. I would recommend that my friends get to know more about 
networking facilities on this campus. 

8 . I would recommend that my friends use this program to 
learn about networking facilities on this campus. 

9. I often felt that I didn't know where to go next when using 
the program. 

1 0. Sometimes I got distracted and forgot why I came to where 
I was. 

1 1. Sometimes I felt like going to sections that I was interested 
in rather than pursuing the answers to the questions. 

12. I liked reading when using the program. 

13. I found it easy to summarize what I had read after reading 
the text in the program. 



4 
4 



4 
4 



5 

5 

5 
5 



6 

6 

6 
6 



7 

7 

7 
7 
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APPENDIX D 

DEFINITIONS OF THE SIX INDICES USED IN CANTER, RIVERS, & STORKS' SCHEME 
AND CHARACTERIZATION OF FIVE DIFFERENT SEARCH STRATEGIES 



Index 



Definition 



Diagram 



Pathiness 



Ringiness 



Loopiness 



Spikiness 



NV/NT 



NV/NS 



A path is a route through the data which does 
not cross any node twice. 



A ring is a route which returns to the node at 
which it starts, such a ring may include other 
rings. 



A loop is a ring which contains no other 
structures and typically stands alone. 



A spike is a route which on the return 
journey retraces exactly the path taken on the 
outward journey. 

The ratio of the number of different nodes 
visited (NV) to the total number of nodes 
available in the system (NT). This ratio is 
always in the range of 0 to 1. 

The ratio of the number of different nodes 
visited (NV) to the total number of visits to 
nodes (NS). This ratio is always in the 
range of 0 to 1 , 




Strategy 


Characteristics 


Description 


Scanning 


High NV/NT 
High spikiness 
Medium loopiness 


Users try to cover a large area without deep depth. 


Browsing 


Medium loopiness 
Medium ringiness 
Medium NV/NS 


Users are happy to go wherever the data takes them 
until their interest is caught. 


Searching 


High spikiness 
Medium loopiness 
Low NV/NS 


Users are motivated to find a particular target and 
therefore create many spikes. 


Exploring 


High pathiness 
High NV/NT 


Users are seeking the extent and nature of the field. 


Wandering 


High ringiness 
Low NV/NS 
Medium NV/NT 


Users amble along and inevitably revisit nodes in an 
unstructured journey. 



